Prior art devices are utilized to delay the lesser delayed signal(s) to match the delay of the more delayed signals. Heretofore, these prior art delay devices have received a Digital Delay Output (DDO) or digital delay signal from the device or system which creates the greater delay, and in response to the DDO operates to adjust a delay mechanism to add an additional delay to the lesser delayed signal(s). For example in television systems a video delay device such as a video synchronizer outputs a DDO which is coupled to a companion audio delay device to cause the audio delay to track the video delay, this operation maintaining the lip sync of the audio with the video. It is generally accepted that the video delay is much more significant than any audio delay and thus audio delays are for the most part ignored. That is in prior art systems the delay imparted to the audio signal from audio signal processing is generally considered insignificant in comparison to the delay imparted to the video signal from video signal processing. While this solution of providing a video DDO to an audio delay which causes the audio delay to match the video delay (with some limitations and errors as described below) has worked well, it nevertheless fails to address a number of heretofore unrecognized problems which are addressed by the present invention.
Prior art problems, and the instant inventive concepts related to solutions for those problems will be taught herein by way of example with respect to television systems having audio and video portions, and in some examples data portions. It will be understood by the person of ordinary skill that the inventive concepts described in the teachings herein are generally applicable to maintaining or correcting the relative timing of related or unrelated signals of many types. As used herein audio and video apply to various forms which audio and video may be processed, carried and/or stored in such as optical, electronic, analog and digital in compressed or uncompressed form.
Of particular trouble is the speed with which the delay of the longer delayed signal can change, and the ability of the tracking delay to adjust to cause a corresponding change of delay in the lesser delayed signal(s), especially when the longer delayed signal experiences a significant change of delay. Significant as used herein with respect to delay change will be understood by the person of ordinary skill in the art as a change of delay which takes place in amount, rate of change or time or combinations thereof, and which depart from the immediately preceding steady state delay conditions. For example significant delay changes include, but are not limited to, changes of the amount of delay either instantly or faster than would be expected from the immediately preceding rate of delay change, changes in the rate of change of delay, the above often being exhibited by the dropping or repeating of one or more block of signal, such as fields or frames of a video signal or GOP of a compressed video signal.
For certainty, as used herein applicant, as his own lexicographer, defines significant when used in association with delay, rate and the like means one (or more) change (larger or smaller) which takes place within a block of signal information which change has a size which exceeds 1% of that block or a rate of change which exceeds 1% of the immediately preceding rate of change, expressed as A per B (i.e. A/B) where A and B are blocks of signal information (not necessarily the same blocks). As one example a significant delay change occurs within a 1 frame block of a television video signal when the delay changes by an amount that is greater than 1% of a frame of television video. A dropped or repeated line, field or frame is another example of such a significant delay change. As yet another example, a delay change which occurs within 1 line, field, frame, GOP, second or minute that is greater than 1% of that 1 line, field, frame, GOP, second or minute is significant.
As still another example one of ordinary skill in the art will understand delay rate changes may be significant delay changes as used herein, and may be expressed for example as a first block rate per second block rate. Such delay rate changes include, but are not limited to, samples per frame, samples per second, seconds per minute, frames per second and frequency changes. A memory writing or reading frequency change (or combination change of the two) of more than 1% is an example of a significant change. As another example, a delay which is getting longer (changing) at a rate of 1 frame per minute changes to getting longer at a rate of 1.3 frames per minute would be a significant delay change.
One of ordinary skill will know the particular blocks of signal information relevant to a significant change to pertain to the nature of the system which causes the delay changes, with the inventive concepts herein being applicable to many different systems and their particular blocks of signals. Note that the block may be expressed as time, memory locations, memory addresses, signal parameters or other such measures as will be known to one of ordinary skill from the teachings herein. For example delays are often expressed in units of time (e.g. seconds and milliseconds) or contiguous groups of samples of data (e.g. lines, fields, frames, bitmaps, GOPs, I frames, B frames, P frames) which one of ordinary skill would know are applicable blocks.
Note here that a steady state condition may be one in which the delay is constant, or is constantly changing at a steady rate. As an example the delay may be slowly and continuously increasing or decreasing (analogous to the rotation of the hands on a motor driven mechanical clock), or may be slowly and incrementally increasing or decreasing (analogous to the stepping of the time display on a digital clock). The delay may take on a significant change instantly (analogous to setting the time on a digital clock) or the rate of change may itself change instantly (analogous to the movement of the hands when setting the time on a mechanical clock). As a further example, consider the motion of the hands on a motor driven mechanical clock when the time is manually changed. When one twists the knob, the hands take a sudden change in rotational velocity which is considered significant as the term is used herein. When the knob is released the hands return to their slow and steady rotation which is considered steady as the term is used herein.
Existing television systems for example can experience significant changes such as instant or very fast changes in the delay of the video signal when one or more fields or frames of video, or group of compressed frames of compressed video (GOPs) are skipped over or repeated in signal processing devices such as video synchronizers. Additionally the rate of change of delay can experience instant or very fast changes such as when the incoming video signal or a synchronizer reference signal changes frequency or is switched to a different video signal having a different frequency.
Additionally video signals which are stored (in any of the various forms which are or will become known to those in the art) are often recalled at an irregular rate. For example a PVR may recall a television program at a speed which is controlled by a viewer who may start, stop, reverse and change speed as the viewer desires. Such changes cause variations in video delay, both in respect of the delay as it relates to the storage of the program but also as it relates of the delay of the video signal which is passed through processing circuitry (for example decompression circuitry) after the signal is read from the storage device. Such variations in delay make it difficult for a tracking audio delay to maintain lip sync of the audio and video portions of the television program.
In another example a video signal is available to a particular point in a system via differing paths having differing delays. For example a video signal from one city may be transmitted to another city via a high quality compression system and channel, with a backup being provided by a lower quality but cheaper compression system and channel. Generally the higher quality compression system will involve more delay to the video signal than the lower quality system because of the extensive signal processing which takes place such as ½ pixel motion prediction, adjustable block sizes and many other improvements to quality video compression which have come about in recent years (and will continue to come about in the future). The two versions of the video signal are available at the particular point, and if for some reason a switch is made from one to the other (such as for example when one falls) an instant delay in the video will often take place, the magnitude of which may be relatively small or may be relatively large. It would not be unexpected to have instant delays in the magnitude of seconds occur in such situations as the quality of video signal processing, such as for example compression, increases in the future.
In still another example, the same scene is captured and conveyed by different sources having different delays, for example a newscaster being televised by two cameras. The same image (the newscaster) may be selected to be passed to additional signal processing (which may itself have an image dependent delay) with the newscaster image experiencing different delays for each camera. Again the delay might be significant and the change in delay can be instant as the selection of one camera or the other is made. Additionally this selection may cause the downstream processing delay to change as the video content is changed.
Note here that in respect to the present invention the inventor defines an image and a scene somewhat differently that those words are commonly utilized in the art. A scene will be understood in this specification and claims to be an electronic representation of a particular thing or set of things. For example a scene of a newscaster captured by a camera, or it may be an electronically created newscaster image. The scene of the newscaster may differ, for example because of the differing angles of two or more cameras capturing the scene, or because the newscaster moves from one frame to the next, or the scene may be precisely the same as for example relatively delayed and undelayed versions thereof. Image will be used in this specification and claims to convey a particular electronic representation which is to be displayed, for example the image from one camera or an electronically generated image. As used herein, unless explicitly defined otherwise via wording or context, image is used to mean a particular image which may be composed of one or more fields or frames, and may be in relatively delayed or undelayed form.
It will be recognized generally that as the quantity of signal processing has, and continues to increase, and the complexity of signal processing systems increases, the amount of instant or fast delay changes a particular signal may experience has also increased. This causes serious problems for prior art tracking delay devices which are expected, but unable, to quickly track these delay changes in order to maintain the relative timing between one or more delayed signal(s) and one or more lesser delayed signal(s).
Consider what happens in a television station which switches from a video signal carrying a scene with a 1 second delay and the scene with a 6 second delay. An instant 5 second delay change is experienced. The accompanying audio delay must be instantly changed by 5 seconds, but the audio delay can not simply be changed by that amount without creating serious artifacts (when speaking of audio herein the inventor intends to reference all types of audio which will be known in the art, for example mono, stereo, 5.1 channel, etc. in optical, electronic, analog, digital and compressed digital form). For example consider a delay where digital audio is written into a circular buffer with one address and read from the buffer with another address. Delay changes are made by instantly changing the read address. If the delay must be shortened, a loss of 5 seconds of audio will be experienced. If the delay is lengthened a silence of 5 seconds in the audio will be experienced. This loss of audio or insertion of silence is clearly unacceptable.
Prior art solutions to overcome the loss of audio or insertion of silence include devices such as described in U.S. Pat. No. 4,313,135 where audio is passed through a shift register with a variable clock is used. The delay is shortened or lengthened by increasing or decreasing the clock frequency and thus the speed at which the audio is clocked through the shift register. This solution causes pitch errors to be created in the audio whenever the delay is changed. In order to minimize the listener's annoyance from the pitch errors, the rate of change of the frequency of the clock must be limited to less than 1%, since most people do not readily detect a 1% pitch error.
Another prior art solution is taught in U.S. Pat. No. 5,920,842. In this invention a circular buffer with a write and two independent read addresses are used and changed to change the delay along with cross fading from a first delayed audio stream to another delayed audio stream, the two streams having slightly different delays. This invention thus can change delays while at the same time compensating for the pitch artifacts which would otherwise be created. Unfortunately the invention of the '842 patent will in some instances still create a noticeable tempo change in the audio.
Another problem with the prior art is that the delay control signal from the video delay is a measure of the current delay of the videos, that is it is a measure of the delay of the video which is being output from the processing which causes the delay. By the time the tracking delay receives the new delay value, that delay has already happened and the compensating delay can not change instantly without creating problems such as those given by example above.